packages/engine/internal/connector/chunk.go (1)

54-69: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

High overlap ratios can cause quadratic memory blow-up.

step = size - overlap can be as small as 1 (overlap = size - 1), which makes sliceChunks emit roughly n chunks, each of length size. For a large document this yields O(n * size) total output memory — e.g. a 100k-character document with chunk_size=1000, chunk_overlap=999 would produce ~100k chunks of ~1000 chars each, well over 100MB. Since this connector feeds directly into ai/embed/kb/upsert batches, this could also translate into a very large downstream batch.

Consider bounding overlap to a fraction of size (e.g., overlap <= size/2) or documenting/capping the maximum chunk count.

♻️ Example validation to cap overlap ratio

 	if overlap >= size {
 		return nil, fmt.Errorf("chunk_overlap (%d) must be less than chunk_size (%d)", overlap, size)
 	}
+	if overlap > size/2 {
+		return nil, fmt.Errorf("chunk_overlap (%d) must not exceed half of chunk_size (%d) to avoid excessive chunk counts", overlap, size)
+	}
 	step := size - overlap

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/engine/internal/connector/chunk.go` around lines 54 - 69, The
sliceChunks helper can generate an excessive number of overlapping chunks when
step is very small, causing memory blow-up downstream in ai/embed and kb/upsert.
Update the chunking logic in sliceChunks (and the overlap-to-step calculation
that feeds it) to cap overlap to a safe fraction of size, such as at most half,
or otherwise enforce a maximum chunk count before emitting windows. Keep the
behavior centered around sliceChunks so the fix is applied where chunk emission
is controlled.